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DETAILED ACTION 

1. Claims 9-16 are pending on this application. 

Information Disclosure Statement 

2. The information disclosure statement filed March 24, 2006 fails to comply with 
the provisions of 37 CFR 1 .97, 1 .98 and MPEP § 609 because a written English 
language translation of the non-English language documents cited has not been 
provided. Also no copy of the document "The Detection of an Object in a Dish Image 
Based on a Constraint from Text Information (no. 2C1-02, June, 2003)", from the papers 
in the 17" Annual Conference of the Japanese Society for Artificial Intelligence has not 
been provided. 

3. It has been placed in the application file, but the information referred to therein 
has not been considered as to the merits. Applicant is advised that the date of any re- 
submission of any item of information contained in this information disclosure statement 
or the submission of any missing element(s) will be the date of submission for purposes 
of determining compliance with the requirements based on the time of filing the 
statement, including all certification requirements for statements under 37 CFR 1.97(e). 
See MPEP § 609.05(a). 

Specification 

4. The disclosure is objected to because of the following informalities: paragraph 2 
describes information contained in a document, and paragraph 3 identifies the 
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document. It seems as if the information contained in these paragraphs should be 
inverted (first identify the document and then describe the information contained 
therein). 

Appropriate correction is required. 

Claim Objections 

5. Claim 9 is objected to because of the following informalities: line 1 7 recites "a 
certainty factor for each of objects that make up..." The claim should read "a certainty 
factor for each object that make up. . ." or "a certainty factor for objects that make up. . ." 
Appropriate correction is required. 

Claim Rejections - 35 USC §112 

6. The following is a quotation of the second paragraph of 35 U.S.C. 112: 

The specification shall conclude with one or more claims particularly pointing out and distinctly 
claiming the subject matter which the applicant regards as his invention. 

Claim 11 is rejected under 35 U.S.C. 112, second paragraph, as being indefinite 
for failing to particularly point out and distinctly claim the subject matter which applicant 
regards as the invention. 

The claim recites: "the feature-quantity template creating means writing, in an 
optional field of the fields which make up the feature-quantity template, as the 
template certainty factor, the greatest certainty factor of the template certainty factors 
which are stored in each field of at least any record of the ingredient record of the 
ingredient table that stores the data which indicates the ingredient that corresponds to 
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the optional field, the cooked-food record of the cooked-food table that stores the data 
which indicates the cooking operation that corresponds to the optional field and the 
cooking-operation record of the cooking-operation table that stores the data which 
indicates the cooking operation that corresponds to the optional field". 

There is no antecedent basis for the fields which make up the feature-quantity 
template, since the claim or the claims from which the claim depends on do not provide 
for fields in the feature quantity template. 

The claim also recites "the greatest certainty factor of the template certainty 
factors which are stored in each field of at least any record of..." It is not clear what 
the intent or scope of this limitation is in the claim, since the claim does not suggest that 
there are different records (or ingredients) and different fields for the records. For 
example, the claim recites: "an ingredient table which is formed by an ingredient 
record that has a field where data..." As written, the claim suggests that there is one 
record (one ingredient) with one field. 

Also note that the claim recites writing as the template certainty factor the 
greatest certainty factor of the template certainty factors which are stored in each field 
of at least any record of the ingredient table, the cooked food table or the cooking 
operation table. It is not clear if there must be a comparison between the tables to 
determine the certainty factor to use as the template certainty factor, if only one of the 
tables is needed to determine the template certainty factor, If only one record from a 
table is needed to determine the greatest certainty factor to use. Moreover, if only one 
record containing one field, as suggested by the claim language, is needed to determine 
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the certainty factor, the intent of this limitation is not clear, since if there is no other 
record to compare that record with, that record will always contain the greatest (and 
only) certainty factor. 

Claim Rejections - 35 USC § 101 

7. 35 U.S.C. 101 reads as follows: 

Whoever invents or discovers any new and useful process, machine, manufacture, or composition of 
matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the 
conditions and requirements of this title. 

Claims 1-14 and 16 are rejected under 35 U.S.C. 101 because the claimed 
invention is directed to non-statutory subject matter. The computer system must set 
forth a practical application of judicial exception to produce a real-world result. Benson, 
409 U.S. at 71-72, 175 USPQ at 676-77. The invention is ineligible because it has not 
been limited to a substantial practical application. 

For a claimed invention to be statutory the claimed invention must produce a 
useful, concrete, and tangible result. As the Supreme Court has made clear, " [a]n idea 
of itself is not patentable," Rubber-Tip Pencil Co. v. Howard, 20 U.S. (1 Wall.) 498, 507 
(1874); taking several abstract ideas and manipulating them together adds nothing to 
the basic equation. In re Warmerdam, 31 USPQ2d 1754 (Fed. Cir. 1994). 

For a claimed invention to be statutory under 35 U.S.C. 101, the claims must 
provide a tangible result, and there must be a practical application, by either: 1) 
transforming (physical thing) or 2) by having the FINAL RESULT (not the steps) achieve 
or produce a useful (specific, substantial, AND credible), concrete (substantially 
repeatable/non-unpredictable), AND tangible (real world/non-abstract) result. 
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In the present case, claim 1 describes a system for recognizing ingredients in a 
cooking operation. However, the claim fails to provide a useful result because the 
claimed subject matter fails to sufficiently reflect at least one practical utility set forth in 
the descriptive portion of the specification. More specifically, while the described 
practical utility (utilities) is (are) directed to estimating a cooking recipe and giving 
guidance to a person on a proper cooking recipe (see paragraph 143 of the 
specification of the instant application), the claimed subject matter relates ONLY to 
recognizing ingredients and cooking operations by the recognitions system. As such, 
the claim is directed to mere abstract manipulation of abstract properties of objects for 
identification. Identification of objects performed by a system, in and of itself, is useless 
in a real world situation absent a particular substantial application for said identification. 
The claims are not limited to sufficiently reflect a substantial practical application for the 
result because they encompass arbitrary identification of objects based on their 
properties, producing an identification that has no specific purpose or use. Claims that 
recite a computer that solely calculates a mathematical formula are not statutory. 

The claims also fail to provide a tangible result because the claimed subject 
matter fails to produce a result that is limited to having real world value rather than a 
result that may be interpreted to be abstract in nature as, for example, a thought, a 
computation, or manipulated data. More specifically, the claimed subject matter 
provides for recognizing objects by a system. This produced result remains in the 
abstract and, thus, fails to achieve the required status of having real world value since it 
is not outputted and provided to a user or to a device in a way as to affect its operation. 
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Claims 10-14 depend from claim 9 and incorporate the same deficiency, and 
furthermore fail to rectify the aforementioned deficiency. 

Claim 16 recites limitations similar to that of claim 9 and is rejected on the same 
basis as set forth above regarding claim 9. 

Claim 11 is rejected under 35 U.S. C. 101 because the claim fails to produce a 
concrete result because the claimed subject matter fails to be limited to the production 
of an assured, repeatable result. More specifically, the claimed subject matter is not 
repeatable because the claim recites writing as the template certainty factor the greatest 
certainty factor of the template certainty factors which are stored in each field of at least 
any record of the ingredient table, the cooked food table or the cooking operation table. 
The claim does not restrict how the system is to select the record to determine the 
greatest certainty factor. Lacking this restriction, the system may select a record 
arbitrarily (any record). Therefore, the system may produce a different result given the 
same inputs, and an assured and repeatable result is not produced each time the 
system is implemented. 

Claim Rejections - 35 USC § 102 

8. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that 
form the basis for the rejections under this section made in this Office action: 

A person shall be entitled to a patent unless - 

(b) the invention was patented or described in a printed publication in this or a foreign country or in public 
use or on sale in this country, more than one year prior to the date of application for patent in the United 
States. 
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9. Claims 9-1 1 and 14-16 are rejected under 35 U.S.C. 102(b) as being anticipated 
by Hamada et al. ("Associating Cooking Video with Related Textbook"; 2000; referred to 
as Hamada). 

Claims 9 and 16 

Hamada anticipates an ingredient cooking-operation recognition system 
(Hamada: pg. 238, section 2; pgs. 239-240, sections 4-4.2) comprising: a sensing 
means for acquiring observation data which includes at least either of an image around 
the hands of a person who performs a cooking operation and an environment sound 
that is produced by the cooking operation (Hamada: abstract; pg. 238, section 2; pgs 
239-240, section 4.1; Examiner's Note (EN): note that only one of these is needed to 
read on the claimed limitation); a feature-quantity template in which various cooking 
operations that are predetermined for various ingredients are stored, together with a 
template certainty factor that is a certainty factor which is predetermined for each 
cooking operation of the ingredients (Hamada: pgs. 238-239, sections 2-3; EN: item 14 
applies. Feature-quantity template and template certainty factor not further defined or 
restricted to any particular structure. The domain specific dictionaries are read as 
templates and the step number or order extracted from the documents are 
predetermined and read as certainty factors); a cooking-flow creating means for, based 
on the observation data which is acquired by the sensing means, calculating an 
observation certainty factor which indicates the certainty factor of at least either of an 
ingredient that is cooked by the person and the cooking operation of the ingredient, and 
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based on this observation certainty factor, creating a cooking flow for a dish which is 
prepared by the person (Hamada: pgs. 237-238, sections 1.2-2; pgs. 239-240, sections 
4-4.2; Fig. 1; EN: item 14 applies. Observation certainty factor not further defined or 
restricted to any particular structure. Image analysis extracts ingredients or objects 
from the video stream and audio analysis detects keywords or objects from the audio 
stream. The shot that has the most common words with a specific step belongs to the 
step (cooking flow)); a primary certainty-factor calculating means for, based on the 
template certainty factor and the observation certainty factor, calculating a primary 
certainty factor which is a certainty factor for each of objects that make up the cooking 
flow which is created by the cooking-flow creating means (Hamada: pgs 238-240, 
sections 2-4.2; Fig. 1 ; EN: item 14 applies. Primary certainty factor not further defined 
or restricted to any particular structure. Linking the documents to a corresponding part 
of the video is read as a primary certainty factor. It is inherent that there must be a 
factor that the computer must consider when linking the document to a portion of the 
video); and a cooking-operation recognizing means recognizing the ingredient and the 
cooking operation that correspond to the observation data which is acquired by the 
sensing means, Based on the primary certainty factor which is calculated by the primary 
certainty-factor calculating means (Hamada: pgs. 239-240, sections 4-4.2; EN: 
analyzing the images from the video to extract objects and actions). 
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Claim 10 

Hamada anticipates a cooking-flow database which stores cooking flows which 
are created in advance for various dishes (Hamada: pg. 237; abstract; pgs. 238-239, 
sections 2-3; pgs 240-241 , sections 4.2-5; EN: the documents collected from the WWW 
cooking program site or the database where recipes can be retrieved); and a relevance- 
factor calculating means for, using mapping, calculating a relevance factor that indicates 
which of the cooking flows that are stored in the cooking-flow database is closest to the 
cooking flow which is created by the cooking-flow creating means (Hamada: pgs 238- 
240, sections 2-4.2; Fig. 1; EN: item 14 applies. No description has been provided in 
the claim as to how the relevance factor is calculated. Determining how common the 
words are between a shot and a step in the preparation within the document to link the 
steps in the document to a part of the video is determining a relevance factor), wherein 
the cooking-operation recognizing means calculates a final certainty factor, based on 
the relevance factor which is calculated by the relevance-factor calculating means and 
the primary certainty factor, updates the feature-quantity template by considering the 
final certainty factor as a new template certainty factor, and based on the feature- 
quantity template, recognizes an ingredient and a cooking operation (Hamada: pgs 237- 
240, sections 1.2-4.2; Fig. 1; EN: detecting the cooking actions and the objects in the 
images from the video). 
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Claim 11 

Hamada anticipates a table storing means for storing at least any one of: an 
ingredient table which is formed by an ingredient record that has a field where data that 
indicates an ingredient is stored and a field where various kinds of data on the 
ingredient are stored together with a template certainty factor that is predetermined for 
the data; a cooked-food table which is formed by a cooked-food record that has a field 
where data that indicates a cooking operation is stored and a field where various kinds 
of data on a change in an ingredient by the cooking operation are stored together with a 
template certainty factor that is predetermined for the data; and a cooking-operation 
table which is formed by a cooking-operation record that has a field where data that 
indicates a cooking operation is stored and a field where various kinds of data on an 
environment sound that is produced by the cooking operation are stored together with a 
template certainty factor that is predetermined for the data (Hamada: pgs. 238-240, 
section 3; EN: item 14 applies. Note that only one of these is needed to be found in the 
prior art to read on the limitation claimed); a cooking-recipe database which stores data 
that indicates an ingredient which is used for various cooking recipes and data that 
indicates a cooking operation for the ingredient (Hamada: pg. 237; abstract; pgs. 238- 
239, sections 2-3; pgs 240-241 , sections 4.2-5; EN: the documents collected from the 
WWW cooking program site or the database where recipes can be retrieved); an 
extracting means for extracting all kinds of ingredients and all kinds of cooking 
operations which are stored in the cooking-recipe database (Hamada: pgs. 238-239, 
sections 2-3; pgs 240-241, sections 4.2-5); and a feature-quantity template creating 
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means for creating, as a feature-quantity template, a table which includes, as items, 
each ingredient that is extracted by the extracting means and each cooking operation 
that is extracted by the extracting means, the feature-quantity template creating means 
writing, in an optional field of the fields which make up the feature-quantity template, 
as the template certainty factor, the greatest certainty factor of the template certainty 
factors which are stored in each field of at least any record of the ingredient record of 
the ingredient table that stores the data which indicates the ingredient that corresponds 
to the optional field, the cooked-food record of the cooked-food table that stores the 
data which indicates the cooking operation that corresponds to the optional field and the 
cooking-operation record of the cooking-operation table that stores the data which 
indicates the cooking operation that corresponds to the optional field (Hamada: pgs. 
238-239, sections 2-3; EN: the field containing the template certainty factor is an 
"optional" field, therefore it is not required). 

Claim 14 

Hamada anticipates the cooking-flow creating means obtains a histogram of hue 
and a histogram of saturation from an image which is acquired by the sensing means, 
and estimates an ingredient by obtaining a correlation between these histograms and a 
color-feature template which is predetermined for each ingredient (Hamada: pgs. 239- 
240, sections 4-4.1; EN: item 14 applies. Performing image analysis using a database 
of colors and textures). 



Application/Control Number: 10/573,387 Page 13 

Art Unit: 2129 

Claim 15 

Hamada anticipates the relevance-factor calculating means estimates a cooking 
recipe based on a relevance factor (Hamada: pg 237, abstract; pgs. 238-240, sections 
2-4.2); and a guidance means is further provided for giving, to the person, guidance on 
the cooking recipe which is estimated by the relevance-factor calculating means 
(Hamada: pg 237, abstract; pgs. 238-240, sections 2-4.2; EN: see especially figs. 3 
and 5). 



Examination Considerations 

10. The claims and only the claims form the metes and bounds of the invention. 
"Office personnel are to give the claims their broadest reasonable interpretation in light 
of the supporting disclosure. In re Morris, 127 F.3d 1048, 105455, 44USPQ2d 1023, 
1027-28 (Fed. Cir. 1997). Limitations appearing in the specification but not recited in the 
claim are not read into the claim. In re Prater, 415 F.2d, 1393, 1404-05, 162 USPQ 541, 
550-551 (CCPA 1969)" (MPEP p 2100-8, c 2, I 45-48; p 2100-9, c 1, I 1-4). The 
Examiner has full latitude to interpret each claim in the broadest reasonable sense. 
Examiner will reference prior art using terminology familiar to one of ordinary skill in the 
art. Such an approach is broad in concept and can be either explicit or implicit in 
meaning. 

1 1 . Examiner's Notes are provided with the cited references to prior art to assist the 
applicant to better understand the nature of the prior art, application of such prior art 
and, as appropriate, to further indicate other prior art that maybe applied in other office 
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actions. Such comments are entirely consistent with the intent and spirit of compact 
prosecution. However, and unless otherwise stated, the Examiner's Notes are not prior 
art but a link to prior art that one of ordinary skill in the art would find inherently 
appropriate. 

12. Unless otherwise annotated, Examiner's statements are to be interpreted in 
reference to that of one of ordinary skill in the art. Statements made in reference to the 
condition of the disclosure constitute, on the face of it, the basis and such would be 
obvious to one of ordinary skill in the art, establishing thereby an inherent prima facie 
statement. 

13. Examiner has cited particular columns and line numbers (or paragraphs) in the 
references applied to the claims above for the convenience of the applicant. Although 
the specified citations are representative of the teachings of the art and are applied to 
specific imitations within the individual claim, other passages and figures may apply as 
well. It is respectfully requested from the Applicant in preparing responses, to fully 
consider the references in their entirety as potentially teaching all or part of the claimed 
invention, as well as the context of the passage as taught by the prior art or disclosed 
by the Examiner. The entire reference is considered to provide disclosure relating to the 
claimed invention 

14. Examiner's Opinion: items 10-13 apply. The claims and only the claims form the 
metes and bounds of the invention. The Examiner has full latitude to interpret each 
claim in the broadest reasonable sense. 
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Conclusion 

15. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

Paris US PGPUB #2002/0171674 

Neuhaus US Patent #5,832,446 

Kudo et al. US PGPUB #2004/0099144 

Hamada et al. "Structural Analysis of Cooking Preparation Steps in Japanese" 
2000, pages 157-164. 

Hamada et al. "Associating Video with Relating Document", 1999, pages 17-20. 

16. Claims 9-16 are rejected. 

Correspondence Information 

1 7. Any inquires concerning this communication or earlier communications from the 
examiner should be directed to Omar F. Fernandez Rivas, who may be reached 
Monday through Friday, between 8:00 a.m. and 5:00 p.m. EST. or via telephone at 
(571) 272-2589 or email omar.fernandezrivas@uspto.gov. 

If you need to send an Official facsimile transmission, please send it to (571) 273-2589. 

If attempts to reach the examiner are unsuccessful the Examiner's Supervisor, 
David Vincent, may be reached at (571) 272-3080. 
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Hand-delivered responses should be delivered to the Receptionist @ (Customer 
Service Window Randolph Building 401 Dulany Street Alexandria, VA 22313), located 
on the first floor of the south side of the Randolph Building. 



Omar F. Fernandez Rivas 
Patent Examiner 
Artificial Intelligence Art Unit 2129 
United States Department of Commerce 
Patent & Trademark Office 



/Omar F. Fernandez Rivas/ 
Examiner, Art Unit 2129 

Monday, March 24, 2008. 



/Joseph P. Hirl/ 

Primary Examiner, Art Unit 2129 



